{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "pandas_dataframe.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.1"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "zwBCE43Cv3PH"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "colab_type": "code",
        "id": "fOad0I2cv569",
        "colab": {}
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YQB7yiF6v9GR"
      },
      "source": [
        "# tf.data を使って pandas の DataFrame をロードする"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Oqa952X4wQKK"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/load_data/pandas_dataframe\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/ja/tutorials/load_data/pandas_dataframe.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/ja/tutorials/load_data/pandas_dataframe.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/ja/tutorials/load_data/pandas_dataframe.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YRulkZPqFXMn",
        "colab_type": "text"
      },
      "source": [
        "Note: これらのドキュメントは私たちTensorFlowコミュニティが翻訳したものです。コミュニティによる 翻訳は**ベストエフォート**であるため、この翻訳が正確であることや[英語の公式ドキュメント](https://www.tensorflow.org/?hl=en)の 最新の状態を反映したものであることを保証することはできません。 この翻訳の品質を向上させるためのご意見をお持ちの方は、GitHubリポジトリ[tensorflow/docs](https://github.com/tensorflow/docs)にプルリクエストをお送りください。 コミュニティによる翻訳やレビューに参加していただける方は、 [docs-ja@tensorflow.org メーリングリスト](https://groups.google.com/a/tensorflow.org/forum/#!forum/docs-ja)にご連絡ください。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BCTbFIJAFEWE",
        "colab_type": "text"
      },
      "source": [
        "このチュートリアルでは、pandas のDataFrameをロードして、`tf.data.Dataset` にデータを読み込む例を示します。\n",
        "\n",
        "このチュートリアルは、クリーブランドクリニック財団（the Cleveland Clinic Foundation for Heart Disease）から提供された、小さな [データセット](https://archive.ics.uci.edu/ml/datasets/heart+Disease) を使っています。このデータセット（CSV）には数百行のデータが含まれています。行は各患者を、列はさまざまな属性を表しています。\n",
        "\n",
        "このデータを使って、患者が心臓病を罹患しているかどうかを判別予測することができます。なお、これは二値分類問題になります。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "iiyC7HkqxlUD"
      },
      "source": [
        "## pandas を使ってデータを読み込む"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "5IoRbCA2n0_V",
        "colab": {}
      },
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "\n",
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "import pandas as pd\n",
        "import tensorflow as tf"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hn4XgljZFEWI",
        "colab_type": "text"
      },
      "source": [
        "heart データセットを含んだCSVをダウンロードします。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "VS4w2LePn9g3",
        "colab": {}
      },
      "source": [
        "csv_file = tf.keras.utils.get_file('heart.csv', 'https://storage.googleapis.com/applied-dl/heart.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "6BXRPD2-xtQ1"
      },
      "source": [
        "pandas を使ってCSVを読み込みます。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "UEfJ8TcMpe-2",
        "colab": {}
      },
      "source": [
        "df = pd.read_csv(csv_file)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "8FkK6QIRpjd4",
        "colab": {}
      },
      "source": [
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "_MOAKz654CT5",
        "colab": {}
      },
      "source": [
        "df.dtypes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dg7Fu8FIFEWU",
        "colab_type": "text"
      },
      "source": [
        "dataframe 内で唯一の `object` 型である `thal` 列を離散値に変換します。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "LmCl5R5C2IKo",
        "colab": {}
      },
      "source": [
        "df['thal'] = pd.Categorical(df['thal'])\n",
        "df['thal'] = df.thal.cat.codes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "s4XA1SNW2QyI",
        "colab": {}
      },
      "source": [
        "df.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "WWRhH6r4xxQu"
      },
      "source": [
        "## `tf.data.Dataset` を使ってデータをロードする"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3yERplFYFEWb",
        "colab_type": "text"
      },
      "source": [
        "`tf.data.Dataset.from_tensor_slices` メソッドを使って、pandas の dataframeから値を読み込みます。\n",
        "\n",
        "`tf.data.Dataset` を使う利点は、シンプルに使えて、かつ、大変効率的なデータパイプラインを構築できることです。詳しくは [loading data guide](https://www.tensorflow.org/guide/data) を参照してください。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "2wwhILm1ycSp",
        "colab": {}
      },
      "source": [
        "target = df.pop('target')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "W6Yc-D3aqyBb",
        "colab": {}
      },
      "source": [
        "dataset = tf.data.Dataset.from_tensor_slices((df.values, target.values))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "chEnp_Swsf0a",
        "colab": {}
      },
      "source": [
        "for feat, targ in dataset.take(5):\n",
        "  print ('Features: {}, Target: {}'.format(feat, targ))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O11hO6nPFEWi",
        "colab_type": "text"
      },
      "source": [
        "`pd.Series` は `__array__` プロトコルを実装しているため、`np.array` や `tf.Tensor` を使うところでは、だいたいどこでも使うことができます。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "GnpHHkpktl5y",
        "colab": {}
      },
      "source": [
        "tf.constant(df['thal'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "9XLxRHS10Ylp"
      },
      "source": [
        "データをシャッフルしてバッチ処理を行います。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "R3dQ-83Ztsgl",
        "colab": {}
      },
      "source": [
        "train_dataset = dataset.shuffle(len(df)).batch(1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bB9C0XJkyQEk"
      },
      "source": [
        "## モデルを作成して訓練する"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "FQd9PcPRpkP4",
        "colab": {}
      },
      "source": [
        "def get_compiled_model():\n",
        "  model = tf.keras.Sequential([\n",
        "    tf.keras.layers.Dense(10, activation='relu'),\n",
        "    tf.keras.layers.Dense(10, activation='relu'),\n",
        "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
        "  ])\n",
        "\n",
        "  model.compile(optimizer='adam',\n",
        "                loss='binary_crossentropy',\n",
        "                metrics=['accuracy'])\n",
        "  return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "ybDzNUheqxJw",
        "colab": {}
      },
      "source": [
        "model = get_compiled_model()\n",
        "model.fit(train_dataset, epochs=15)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "d6V_6F_MBiG9"
      },
      "source": [
        "## 特徴列の代替"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vIqjDd0PFEWu",
        "colab_type": "text"
      },
      "source": [
        "モデルへの入力に辞書型データを渡すことは、 `tf.keras.layers.Input` におなじ型の辞書を作成し、何らかの前処理を適用して、[functional api](../../guide/keras/functional.ipynb) を使ってスタッキングすることと同様に、簡単に行うことができます。これを [特徴列](../keras/feature_columns.ipynb) の替わりに使うことができます。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "FwQ47_WmOBnY",
        "colab": {}
      },
      "source": [
        "inputs = {key: tf.keras.layers.Input(shape=(), name=key) for key in df.keys()}\n",
        "x = tf.stack(list(inputs.values()), axis=-1)\n",
        "\n",
        "x = tf.keras.layers.Dense(10, activation='relu')(x)\n",
        "output = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n",
        "\n",
        "model_func = tf.keras.Model(inputs=inputs, outputs=output)\n",
        "\n",
        "model_func.compile(optimizer='adam',\n",
        "                   loss='binary_crossentropy',\n",
        "                   metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_a6DP7VnFEWz",
        "colab_type": "text"
      },
      "source": [
        "`tf.data` を使うときに、pandas の DataFrame の列構造を保持する一番簡単な方法は、DataFrame を辞書型データに変換して、先頭を切り取ることです。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "wUjRKgEhPZqK",
        "colab": {}
      },
      "source": [
        "dict_slices = tf.data.Dataset.from_tensor_slices((df.to_dict('list'), target.values)).batch(16)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "WWRaiwxeyA9Z",
        "colab": {}
      },
      "source": [
        "for dict_slice in dict_slices.take(1):\n",
        "  print (dict_slice)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "8nTrfczNyKup",
        "colab": {}
      },
      "source": [
        "model_func.fit(dict_slices, epochs=15)"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}